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Purpose 

The purpose of this study is to disseminate the results of recent advances in statistical power 
analyses with regard to multilevel mediation and its implementation in the PowerUp!-Mediator 
software (Dong, Kelcey, Spybrook, & Maynard, 2016; Kelcey, Dong, Spybrook, & Shen, in 
review; Kelcey, Dong, Spybrook, & Cox, in review;). We first focus on the conceptual and 
statistical differences among common asymptotic, component-wise, and resampling-based tests 
of mediation as well as their performance in different contexts. We then introduce newly derived 
power formulas and delineate the statistical and substantive interpretation of the parameters that 
govern power in studies of multilevel mediation. Third, we outline reasonable values for these 
parameters across different education contexts using recent empirical compilations of values of 
these parameters. Finally, we demonstrate the use of the PowerUp!-Mediator software along with 
the formulas and parameter values to plan studies. 


Background 

Designs that facilitate inferences concerning both the total and indirect effects of a treatment 
potentially offer a more holistic description of interventions because they can complement ‘what 
works’ questions with the comprehensive study of the causal connections implied by substantive 
theories. Mapping the sensitivity of designs to detect these effects is of critical importance 
because it directly governs the types of evidence researchers can bring to bear on theories of 
action under realistic sample sizes. In this study, we review recent research deriving closed-form 
expressions to estimate the power to detect causally-defined individual, contextual, and 
cumulative indirect effects in two-level group-randomized studies. We unpack these formulas 
under the purview of typical multilevel mediation models and anchor their interpretation in the 
potential outcomes framework. The framework provides power analysis formulas and software 
that reduce calculations to simple functions of the primary path coefficients (e.g., treatment- 
mediator and mediator-outcome relationships) and common summary statistics (e.g., intraclass 
correlation coefficients). Probing these formulas suggests that group-randomized designs will 
typically be well-powered to detect individual indirect effects and can be well-powered to detect 
contextual and cumulative indirect effects when carefully planned. 


Illustration 

To briefly demonstrate the practical use of the formulas and software, let us consider a two-level 
study with students nested within schools. The goal of the study is to understand the extent to 
which a school-level program improves a student-level outcome through a student-level 
mediator. For instance, assume we are studying an intervention that has been randomly assigned 
to schools and is designed to improve student mental health (outcome) by improving student 
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knowledge of positive mental health behaviors (mediator). In addition, researchers suspect that a 
secondary pathway involves the intervention impacting the outcome through changes in the 
collective school-level knowledge base of students. As a result, the theory of action suggests that 
the intervention improves student mental health by both improving individual and collective 
student knowledge. 


In planning this study, let us assume that we intend to sample 50 students per school (7), expect 
intraclass correlation coefficients for the outcome and mediator of 0.10 (p), and anticipate that 
the covariates collected will explain about 10% of the variance in the mediator at both levels and 
25% of the variance in the outcome at both levels. Let us further assume that we anticipate a total 
or average treatment effect of 0.40—of which only a small 0.05 (12.5%) flows through the 
individual-level mediator while another 0.15 (37.5%) passes through the group-level mediator. 
Let us hypothesize that the decomposition of these indirect effects happens as follows: the 
projected difference in mediator values between treatment and control conditions is 
approximately 0.5 standard deviations (a), and the unique individual- (b;) and group -level (b2) 
mediator-outcome standardized associations are 0.10 and 0.30 respectively (and assume 
treatment-mediator interactions add/subtract 0.025 at both levels for the treatment/control group). 
That is, let a=0.5, b=0.10, b2=0.30, c’=0.20, 6)/2= 62/2=0.025, R°,, = Rie =0.10, and 


M 


Li 
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Riu = Revs = 0.25. Given these guidelines, we wish to identify a school-level sample size that 


would provide an 80% chance of detecting a total cross-level indirect effect as small as 0.0625 
(0.5(0.10+.025)) assuming a type one error rate of 0.05. 


Figure | outlines the software module selection and parameter value input for power analyses for 
indirect effects in two-level cluster-randomized trials (1.e., cluster random assignment designs 
[CRA]). The top panel outlines the levels of the treatment, mediator, and outcome (e.g., 2-1-1 
refers to a study in which the treatment has been assigned to clusters and the mediator and 
outcome are measured at the individual-level). In the bottom panel, users input the selected 
values for the parameters (to be discussed in detail). In turn, Figure 2 plots the power formulas 
tracking the power to detect the cross-level (individual) indirect effects, school-level (contextual) 
indirect effects and overall (individual and contextual) indirect effects. 


In Figure 2, we display the power for the Sobel (solid curves), joint (dashed curves), and Monte 
Carlo interval tests (dotted curves) for the overall (black), school-level (light gray) and cross- 
level (dark gray) indirect effects as a function of group-level sample size (nz). For the current 
example, our analyses suggest that sampling even just 18 schools (9 treatment, 9 control) would 
provide a sufficient power to detect the cross-level indirect effect. For the overall indirect effect 
we would need between 64 and 72 schools and for the school indirect effect we would need 
about 126 schools. 
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Figure | 
Software module selection and parameter value input 


PowerUp!-Mediator to Detect Mediator Effects in 2-Level CRTs: Models and 
Corresponding Worksheets 


Number of Total 


: Model Number | Level of Mediator | Power Calculation 
Levels of Clustering 


Study Design 


Cluster Random Assignment Designs (Level of Assignment = Level of Analysis) 


Simple Cluster 
Random Assignment CRA2 1 1 
(CRA), or Cluster — 
Randomized Trials 
(CRTs) CRA2 2 1 


Assumptions Comments 
Probability of a Type | error 


Alpha Level (a) 

Effect Size: 3 

Effect Size: «’ 

Effect size (standardized coefficient] of the mediator*treatment on the level-2 outcome in path b (or path c’). 


Intraclass Correlation (0 } Proportion of variance in outcome that is between clusters 
msi Proportion of variance in level-1 outcomes [‘’] explained by level-1 covariates 


Fite 
n (Average Cluster Size] Mean number of level-1 units per level-2 cluster (harmonic mean recommended) 


J{Sample Size [# of Clusters] Toal number of level-2 units 
Noncentrality Parameter for PIE 2.49 for Sobel test 


Noncentrality Parameter for TIE 2.62 For Sobel test 
Noncentrality Parameter for path a 2.78 | Test of the treatment effect an the mediator (T--> M] 
Noncentrality Parameter for path b- 5.61 Test of the effect of the mediator on the outcome (M--> Y) for the control group 


Noncentrality Parameter for path b+ Test of the effect of the mediator on the outcome (M--> Y) for the treatment group 


e125 | Pure incivect Effect (PIE) 
pe 0175 Total ncirect Effect (TIE 


Effect size (Cohen's d) of treatment on the mediator in path a. 


Effect size [standardized coefficient) of the mediator on the level-2 outcome in path b [or path c'). 


Effect size [Cohen's d)J of the direct effect of treatrnent on the level-2 outcome in path c’. 


Proportion of variance in level-2 outcome [‘Y] means explained by level-2 covariates 


Proportion of variance in the mediator explained by level-2 covariates 


Power (1-6) for PIE: MC 0.818 Statistical power to detect the Pure Indirect Effect (PIE) for two-tailed test using Monte Carlo simulation 
Power (1-6) for TIE: MC 0.818 Statistical power to detect the Total Indirect Effect (TIE) for two-tailed test using Monte Carlo simulation 
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Figure 2 

Power for the Sobel (solid curves), joint (dashed curves), and Monte Carlo interval tests (dotted 
curves) in detecting the total overall (black), group-level (light gray) and cross-level (dark gray) 
indirect effects as a function of group-level sample size (n2) assuming 1/=50, pm= py = 0.10, 
Riu = Rye =0.10, Ru = Riv =0.25 , a=0.5, bj=0.10, b2=0.30, c’=0.20, 67/2= 62/2=0.025. 


Power 


50 100 150 200 


Group Sample Size 


SREE Spring 2017 Conference Symposium: Paper #4 Abstract 4-5 


